Omnidirectional image processing device and omnidirectional image processing method

ABSTRACT

An omnidirectional image processing apparatus is provided that enables the visibility of a specified monitoring object in an omnidirectional image to be improved while maintaining the continuity of that omnidirectional image. An omnidirectional image processing apparatus ( 400 ) is an apparatus that performs image processing on an omnidirectional image, and has a monitoring object specification section ( 405 ) that specifies a monitoring object photographed in the omnidirectional image, an image rotating section ( 406 ) that rotates the omnidirectional image so that the position of a specified monitoring object becomes upper-central, and a center position moving section ( 407 ) that moves the center position of the omnidirectional image rotated by the image rotating section ( 406 ) downward by means of warping processing.

TECHNICAL FIELD

The present invention relates to an omnidirectional image processingapparatus and omnidirectional image processing method that perform imageprocessing on an omnidirectional image.

BACKGROUND ART

An omnidirectional camera provides for an omnidirectional image to bephotographed in a single shot by using a camera configured by combininga special optical system such as a fisheye lens, spherical mirror, orthe like with an ordinary camera (see Non-Patent Literature 1, forexample). Omnidirectional cameras are used in a wide range of fields,including monitoring systems. However, in an omnidirectional image thereis major image distortion of people, objects, and suchlike individualphotographic subjects (hereinafter referred to as “monitoring objects”),making it difficult to grasp the appearance of individual monitoringobjects.

Thus, technologies have hitherto been proposed that perform imageprocessing on an omnidirectional image (see Patent Literature 1 andPatent Literature 2, for example). The technology described in PatentLiterature 1 converts an omnidirectional image to an annularimage—specifically, to an image displayed within an area between twolarger and smaller ellipses with different center positions. By means ofthis technology, it is possible to easily grasp the positionalrelationship of individual monitoring objects. The technology describedin Patent Literature 2 performs coordinate conversion of anomnidirectional image to a panoramic landscape-format image, andperforms distortion correction by clipping an image of a specifiedmonitoring object. By means of this technology, it is possible togenerate an image with little individual monitoring object imagedistortion from an omnidirectional image.

CITATION LIST Patent Literature PTL 1

-   Japanese Patent Application Laid-Open No. 2003-132348

PTL 2

-   Japanese Patent Application Laid-Open No. 2008-48443

PTL 3

-   Japanese Patent Application Laid-Open No. HEI11-331654

Non-Patent Literature NPL 1

-   YAMAZAWA Kazumasa, YAGI Yasushi, YACHIDA Masahiko, “Visual    Navigation with Omnidirectional Image Sensor HyperOmni Vision,”    Transactions of IEICE D-II, IEICE, 25 May 1996, Vol. J79-D-II, p.    698-707

SUMMARY OF INVENTION Technical Problem

However, with the technologies described in Patent Literature 1 andPatent Literature 2, it may be difficult to grasp the detailed situationof monitoring objects in a wide range, or to intuitively grasp theambient situation of a monitoring object.

With the technology described in Patent Literature 1, the reason forthis is that, if a monitoring object is present in the narrower of areasbetween two larger and smaller ellipses with different center positions,an image of a monitoring object and monitoring object surroundings isreduced. And with the technology described in Patent Literature 2, thereason is that only a specified monitoring object image part is clippedand displayed, and omnidirectional image continuity is lost.

It is therefore an object of the present invention to provide anomnidirectional image processing apparatus and method thereof thatimprove the visibility of a specified monitoring object in anomnidirectional image while maintaining the continuity of thatomnidirectional image.

Solution to Problem

An omnidirectional image processing apparatus of the present inventionperforms image processing on an omnidirectional image, and has: amonitoring object specification section that specifies a monitoringobject photographed in the omnidirectional image; an image rotatingsection that rotates the omnidirectional image so that the position ofthe specified monitoring object becomes upper-central; and a centerposition moving section that moves the center position of theomnidirectional image rotated by the image rotating section downward bymeans of warping processing.

An omnidirectional image processing method of the present inventionperforms image processing on an omnidirectional image, and has: a stepof specifying a monitoring object photographed in the omnidirectionalimage; a step of rotating the omnidirectional image so that the positionof the specified monitoring object becomes upper-central; and a step ofmoving the center position of the rotated omnidirectional image downwardby means of warping processing.

Advantageous Effects of Invention

The present invention enables the visibility of a specified monitoringobject in an omnidirectional image to be improved while maintaining thecontinuity of that omnidirectional image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a system configuration diagram showing an installation exampleof a monitoring system that includes an omnidirectional image processingapparatus according to an embodiment of the present invention;

FIG. 2 is a drawing showing an example of an omnidirectional image inthis embodiment;

FIG. 3 is a block diagram showing the configuration of anomnidirectional image processing apparatus according to this embodiment;

FIG. 4 is a drawing showing an example of a frame time correspondencetable in this embodiment;

FIG. 5 is a drawing showing an example of monitoring object positioninformation in this embodiment;

FIG. 6 is a drawing showing an example of monitoring object positionmanagement information in this embodiment;

FIG. 7 is a flowchart showing an image processing operation in thisembodiment;

FIG. 8 is a drawing showing a first example of a monitoring objectspecification screen in this embodiment;

FIG. 9 is a drawing showing a second example of a monitoring objectspecification screen in this embodiment;

FIG. 10 is a drawing showing a third example of a monitoring objectspecification screen in this embodiment;

FIG. 11 is a flowchart showing an example of image rotation processingin this embodiment;

FIG. 12 is a drawing showing an example of an omnidirectional image ininitial-state in this embodiment;

FIG. 13 is a drawing showing an example of definition of a specifiedmonitoring object area in this embodiment;

FIG. 14 is a drawing showing an example of monitoring object areainformation in this embodiment;

FIG. 15 is a drawing for explaining an example of a rotation amountcalculation method in this embodiment;

FIG. 16 is a drawing showing an example of a post-rotation image in thisembodiment;

FIG. 17 is a flowchart showing an example of center position movementprocessing in this embodiment;

FIG. 18 is a drawing showing an example of pre-movement image areadivision in this embodiment;

FIG. 19 is a drawing showing an example of post-movement image areadivision in this embodiment;

FIG. 20 is a drawing showing an example of the state of each position ina post-ellipsification image in this embodiment;

FIG. 21 is a flowchart showing an example of distortion correctionprocessing in this embodiment;

FIG. 22 is a drawing showing an example of pre-distortion-correctionimage area division in this embodiment;

FIG. 23 is a drawing showing an example of post-distortion-correctionimage area division in this embodiment;

FIG. 24 is a drawing showing another example of definition of aspecified monitoring object area in this embodiment;

FIG. 25 is a drawing showing another example of an omnidirectional imagein initial-state in this embodiment; and

FIG. 26 is a drawing showing another example of a post-rotation image inthis embodiment.

DESCRIPTION OF EMBODIMENT

Now, an embodiment of the present invention will be described in detailwith reference to the accompanying drawings.

FIG. 1 is a system configuration diagram showing an installation exampleof a monitoring system that includes an omnidirectional image processingapparatus according to an embodiment of the present invention. Thisembodiment is an example in which the present invention is applied to amonitoring system for monitoring workers in a factory.

As shown in FIG. 1, monitoring system 100 has omnidirectional camera 300installed on the ceiling or the like of monitored area 200 in a factory,facing the floor, and omnidirectional image processing apparatus 400connected to omnidirectional camera 300 so as to be able to communicatetherewith.

Omnidirectional camera 300 is a camera capable of photographing anomnidirectional image, and here, a camera that uses a special mirrorsuch as described in the Non-Patent Literature will be described as anexample. Omnidirectional camera 300 photographs monitored area 200 inwhich workers 210 are working while the factory is operating, andtransmits moving image data that is time series data of a photographedomnidirectional image to omnidirectional image processing apparatus 400.

FIG. 2 is a drawing showing an example of an omnidirectional imagephotographed by omnidirectional camera 300.

As shown in FIG. 2, the shape of image 501 of moving image datatransmitted to omnidirectional image processing apparatus 400 byomnidirectional camera 300 is square, whereas the shape ofomnidirectional image 502 in which monitored area 200 is photographed iscircular. This is because the maximum cross-section with respect to theoptical axis of a mirror used in this embodiment is circular. Therefore,non-photographed parts 503 occur in the four corners of moving imagedata image 501. Also, since omnidirectional camera 300 according to thisembodiment has a small reflecting mirror installed on the incident lightaxis, circular blind area 504 may occur in the center of omnidirectionalimage 502. The reason for the occurrence of a blind area is explained inNon-Patent Literature 1 and Patent Literature 3, and therefore anexplanation thereof will be omitted here.

Here, an omnidirectional image photographed by omnidirectional camera300 is a downward-view image—that is, an image whose center is directlybelow the omnidirectional camera. Therefore, in image 505 of a workerstanding on the floor of monitored area 200, the head is oriented towardthe circumference of omnidirectional image 502, and the lower body isoriented toward the center of omnidirectional image 502. Square-shapedmoving image data image 501 corresponds to one frame, but below,circle-shaped omnidirectional image 502 including blind area 504 isdescribed as an image processing object.

Omnidirectional image processing apparatus 400 in FIG. 1 is, forexample, a personal computer. Omnidirectional image processing apparatus400 performs image processing on an omnidirectional image received fromomnidirectional camera 300, and converts this image to an image that isused in flow line analysis or the like with the aim of improving thework efficiency of workers in a factory. Specifically, omnidirectionalimage processing apparatus 400 performs processing that rotates anomnidirectional image, and warping processing or the like that moves thecenter position of an omnidirectional image. Then omnidirectional imageprocessing apparatus 400 displays an omnidirectional image on which suchprocessing has been performed.

Image rotation processing is processing that rotates an omnidirectionalimage so that a specified monitoring object is positioned in the uppercenter of the omnidirectional image.

Warping processing is assumed to refer to deforming an image byexecuting nonlinear geometric transformations on division areas intowhich an image is divided so that an adjacency relationship between thedivision areas is not destroyed. Warping processing here is, forexample, processing that moves the center position of an omnidirectionalimage.

Omnidirectional image processing apparatus 400 of the present inventioncan display an omnidirectional image that enables the overall appearanceof monitored area 200 to be grasped. That is to say, omnidirectionalimage processing apparatus 400 can convert an omnidirectional image toan image in which there is little distortion of a specified monitoringobject image area while maintaining the continuity of that image, anddisplay the converted image. It is also possible for omnidirectionalimage processing apparatus 400 to perform enlarged display of an imagearea that includes a monitoring object. By this means, a user, such as asystem administrator or the like, can recognize the detailed situationof a specified monitoring object (here, worker 210), and the ambientsituation, by viewing a converted image.

The configuration of omnidirectional image processing apparatus 400 willnow be described.

FIG. 3 is a block diagram showing the configuration of omnidirectionalimage processing apparatus 400.

In FIG. 3, omnidirectional image processing apparatus 400 has movingimage input section 401, moving image storage section 402, monitoringobject position information input section 403, monitoring objectposition information storage section 404, monitoring objectspecification section 405, image rotating section 406, center positionmoving section 407, ellipsification section 408, distortion correctionsection 409, output image storage section 410, and image output section411.

Moving image input section 401 inputs moving image data. Here, movingimage input section 401 receives moving image data of an omnidirectionalimage in which monitored area 200 has been photographed fromomnidirectional camera 300, and encodes the received moving image data.When performing encoding, moving image input section 401 assigns a frameID (a value that is incremented by 1 each time) to each moving imagedata, and also assigns the encoding time as a time stamp. Moving imageinput section 401 outputs encoded moving image data to moving imagestorage section 402.

Moving image storage section 402 stores moving image data input frommoving image input section 401 in a moving image format comprising aplurality of frames, such as the H.264 format. Then moving image storagesection 402 receives a directive from monitoring object specificationsection 405, and outputs a frame (omnidirectional image) with a frame IDspecified by monitoring object specification section 405 to imagerotating section 406. Moving image storage section 402 also holds aframe time correspondence table in a state enabling referencing bymonitoring object position information storage section 404. The frametime correspondence table comprises information indicating acorrespondence relationship between a frame ID and a time at which thecorresponding frame was photographed (a time input from moving imageinput section 401).

Monitoring object position information input section 403 inputsmonitoring object position information of each monitoring object atpredetermined intervals (for example, every 0.5 second), and outputsinput monitoring object position information to monitoring objectposition information storage section 404. Monitoring object positioninformation is information indicating a time, and two-dimensionalposition coordinates on an omnidirectional image of a representativepoint of a monitoring object present in monitored area 200 at that time(hereinafter referred to as an “in-screen position”), in a mutuallyassociated fashion. A representative point of a monitoring object is,for example, a position of a wireless tag when acquired using a wirelesstag, or the center point of a monitoring object image area when acquiredby means of image recognition performed on an omnidirectional image.

Monitoring object position information storage section 404 generatesmonitoring object position management information based on a frame timecorrespondence table input from moving image storage section 402 andmonitoring object position information input from monitoring objectposition information input section 403, and stores that monitoringobject position management information. Monitoring object positioninformation storage section 404 stores this information in a stateenabling referencing by monitoring object specification section 405.

Monitoring object specification section 405 specifies an omnidirectionalimage (frame) to be displayed from among the moving image data stored bymoving image storage section 402. Monitoring object specificationsection 405 also specifies a monitoring object to be displayed in aparticularly easy-to-see fashion from among monitoring objectsphotographed in an omnidirectional image. Details of the actual frameand monitoring object specification methods will be given later herein.Monitoring object specification section 405 causes moving image storagesection 402 to output an omnidirectional image of a specified frame toimage rotating section 406.

Monitoring object specification section 405 detects an in-screenposition of a specified monitoring object in a specified frame. Thenmonitoring object specification section 405 acquires a detectedspecified monitoring object in-screen position (hereinafter referred toas “specified monitoring object position”), and outputs this in-screenposition to image rotating section 406. Monitoring object specificationsection 405 is here assumed to receive specified frame and specifiedmonitoring object selections from a user, using a display apparatus suchas a liquid crystal display and an input apparatus such as a keyboardand/or mouse (neither of which is shown).

Based on a specified monitoring object position, image rotating section406 performs the above image rotation processing on an omnidirectionalimage of a specified frame input from moving image storage section 402.Image rotating section 406 also extracts an image area occupied by aspecified monitoring object within an omnidirectional screen(hereinafter referred to as “specified monitoring object area”). Thenimage rotating section 406 outputs an omnidirectional image after imagerotation processing has been performed, and a specified monitoringobject position and specified monitoring object area (informationcombining the two is referred to below as “monitoring object positioninformation”), to center position moving section 407.

Below, an omnidirectional image immediately before image rotationprocessing is performed is referred to as a “pre-rotation image,” and anomnidirectional image immediately after image rotation processing hasbeen performed is referred to as a “post-rotation image.”

Center position moving section 407 performs warping processing thatmoves the center position of a post-rotation image downward (hereinafterreferred to as “center position movement processing”) on a post-rotationimage input from image rotating section 406. Then center position movingsection 407 outputs a post-rotation image after center position movementprocessing has been performed, and monitoring object positioninformation, to ellipsification section 408.

Below, an omnidirectional image immediately before center positionmovement processing is performed is referred to as a “pre-movementimage,” and an omnidirectional image immediately after center positionmovement processing has been performed is referred to as a“post-movement image.”

Ellipsification section 408 performs processing (hereinafter referred toas “ellipsification processing”) on a post-movement image input fromcenter position moving section 407 whereby the shape of that image ischanged from a circle to an ellipse by means of a linear projectivetransformation. Then ellipsification section 408 outputs a post-movementimage after ellipsification processing has been performed, andmonitoring object position information, to distortion correction section409.

Below, an omnidirectional image immediately before ellipsificationprocessing is performed is referred to as a “pre-ellipsification image,”and an omnidirectional image immediately after ellipsificationprocessing has been performed is referred to as a “post-ellipsificationimage.”

Distortion correction section 409 performs processing that correctsomnidirectional image specific distortion (hereinafter referred to as“distortion correction processing”) for a specified monitoring objectarea of a post-ellipsification image input from ellipsification section408. Then distortion correction section 409 outputs apost-ellipsification image after distortion correction processing hasbeen performed, and monitoring object position information, to outputimage storage section 410.

Below, an omnidirectional image immediately before distortion correctionprocessing is performed is referred to as a “pre-distortion-correctionimage,” and an omnidirectional image immediately after distortioncorrection processing has been performed is referred to as a“post-distortion-correction image.”

Output image storage section 410 stores a post-distortion-correctionimage input from distortion correction section 409.

Image output section 411 reads a post-distortion-correction image storedby output image storage section 410 as a final omnidirectional image,and outputs this image to a display apparatus such as a liquid crystaldisplay.

Omnidirectional image processing apparatus 400 of this kind can beimplemented by means of a CPU (central processing unit), a storagemedium such as ROM (read only memory) that stores a control program,working memory such as RAM (random access memory), a storage medium suchas a hard disk for storing various kinds of data, and so forth. In thiscase, the functions of the above sections are implemented by executionof the control program by the CPU.

Omnidirectional image processing apparatus 400 having this kind ofconfiguration can display an image of a specified monitoring object in astate closer to a view when the specified monitoring object is viewed inreal space in an omnidirectional image, while maintaining the continuityof that omnidirectional image. Also, omnidirectional image processingapparatus 400 can perform enlarged display of an image area thatincludes a monitoring object. That is to say, omnidirectional imageprocessing apparatus 400 can improve the visibility of a monitoringobject that a user wishes to focus on in an omnidirectional image, whilemaintaining the continuity of that omnidirectional image.

The overall operation of omnidirectional image processing apparatus 400will now be described.

Here, it is assumed that, after performing an information storageoperation, omnidirectional image processing apparatus 400 performs animage processing operation that processes an omnidirectional image basedon stored moving image data and monitoring object position managementinformation. Here, an information storage operation is assumed to be anoperation that stores moving image data and monitoring object positionmanagement information. First, an information storage operation will bedescribed.

Omnidirectional image processing apparatus 400 first inputs moving imagedata photographed by omnidirectional camera 300 via moving image inputsection 401, and stores this data in moving image storage section 402.Moving image storage section 402 generates a frame time correspondencetable for stored moving image data.

FIG. 4 is a drawing showing an example of a frame time correspondencetable.

As shown in FIG. 4, frame time correspondence table 610 holdsomnidirectional image photographed time 611 and frame ID 612 of thatomnidirectional image in a mutually associated fashion. Thephotographing cycle is fixed, and one frame corresponds to onephotographed time.

Here, it is assumed that the time from photographing of anomnidirectional image until input of a corresponding frame to movingimage storage section 402 is extremely short. Therefore, moving imagestorage section 402 uses an input time itself as frame ID 612. When thetime required until a corresponding frame is input to moving imagestorage section 402 after an omnidirectional image is photographed isconsidered, moving image storage section 402 can use a time upstream bythat time period from the input time as photographed time 611.

Monitoring object position information input section 403 inputsmonitoring object position information of a photographed monitoringobject for a monitoring object present in monitored area 200—that is, anomnidirectional image photographed by omnidirectional camera 300.Monitoring object position information includes a time and a position inan omnidirectional image of each monitoring object. Monitoring objectposition information input section 403 outputs input monitoring objectposition information to monitoring object position information storagesection 404.

FIG. 5 is a drawing showing an example of monitoring object positioninformation.

As shown in FIG. 5, monitoring object position information 620 containsdetection time 621 at which monitoring object position detection wasperformed, monitoring object ID 622, and detected monitoring objectin-screen position 623, in a mutually associated fashion. In-screenposition 623 indicates a position in moving image data, and isrepresented, for example, by coordinate values when an xy coordinatesystem is defined with the bottom-left corner of moving image data asthe origin, as described later herein.

Monitoring object position information can be obtained by means ofpositioning using a wireless tag, image recognition performed on anomnidirectional image, and so forth.

In the case of positioning using a wireless tag, monitoring objectposition information input section 403, for example, stores beforehand acorrespondence relationship between a three-dimensional position in realspace (monitored area 200) and a two-dimensional position in anomnidirectional image photographed by omnidirectional camera 300. Thenmonitoring object position information input section 403 acquires IDinformation of each monitoring object (hereinafter referred to as“monitoring object ID”) and a three-dimensional position in real spaceof a representative point of each monitoring object from wireless tagsattached to each monitoring object at predetermined intervals (forexample, every 0.5 second). Next, monitoring object position informationinput section 403 converts an acquired three-dimensional position to atwo-dimensional position in an omnidirectional image. At this time,monitoring object position information input section 403 may use an IDof each wireless tag directly as a monitoring object ID.

When image recognition is used, monitoring object position informationinput section 403, for example, stores beforehand image features such afacial feature amount and hat color of each monitoring object.Monitoring object position information input section 403 extracts animage area occupied by each monitoring object in an omnidirectionalimage for each frame corresponding to a predetermined interval. Thenmonitoring object position information input section 403 performs facerecognition, hat color recognition, or the like, and acquires amonitoring object ID of each monitoring object, and also acquires arepresentative position in the omnidirectional image of each image areaas a two-dimensional position of that monitoring object.

Based on monitoring object position information input from monitoringobject position information input section 403, monitoring objectposition information storage section 404 generates monitoring objectposition management information, and stores the generated monitoringobject position management information.

FIG. 6 is a drawing showing an example of monitoring object positionmanagement information.

As shown in FIG. 6, monitoring object position management information630 contains, associated with frame ID 631, monitoring object ID 632 ofa monitoring object for which there is a possibility of having beenphotographed in an omnidirectional image of this frame, and in-screenposition 633 thereof.

Monitoring object position information storage section 404 generatesmonitoring object position management information 630 as describedbelow, for example. Monitoring object position information storagesection 404 references frame time correspondence table 610 stored inmoving image storage section 402 (see FIG. 4). Then monitoring objectposition information storage section 404 searches for an interval thatincludes detection time 621 of monitoring object position information620 (see FIG. 5) among photographed time 611 intervals contained inframe time correspondence table 610. A photographed time 611 intervalmeans a period from the photographed time of that frame until thephotographed time of the next frame. Then monitoring object positioninformation storage section 404 mutually associates frame ID 612corresponding to a relevant interval shown in monitoring object positionmanagement information 630, and monitoring object ID 622 and in-screenposition 623 of monitoring object position information 620 shown in FIG.5.

For example, as shown in FIG. 5, monitoring object position information620 with monitoring object ID 622 “5” and in-screen position 623 “(780,550)” associated with detection time 621 “2010/1/13 9:34:55 02” isinput. Also, it is assumed that frame time correspondence table 610 withframe ID 612 “10” associated with an interval that includes “2010/1/139:34:55 02” has been stored, as shown in FIG. 4.

In this case, generated monitoring object position managementinformation 630 has contents associating monitoring object ID 632 “5”and in-screen position 633 “(780, 550)” with frame ID 631 “10,” as shownin FIG. 6.

By means of the above information storage operation, a state isestablished in which moving image data and monitoring object positionmanagement information are stored in moving image storage section 402and monitoring object position information storage section 404respectively. In this state, it is possible for omnidirectional imageprocessing apparatus 400 to perform an image processing operation. Forexample, after completing monitoring object position managementinformation storage of one day's worth of monitored area 200 movingimage data, omnidirectional image processing apparatus 400 performs animage processing operation using this data the following day.

An image processing operation will now be described.

FIG. 7 is a flowchart showing an image processing operation ofomnidirectional image processing apparatus 400.

First, in step S1000, monitoring object specification section 405decides a specified monitoring object and specified frame based onmonitoring object position management information stored in monitoringobject position information storage section 404 (see FIG. 6). Aspecified monitoring object and specified frame may be decidedarbitrarily by a user, or may be decided automatically based on apredetermined rule. The contents of a predetermined rule may be, forexample, that a specific worker is taken as a specified monitoringobject, and a frame of a time period in which that worker is at work istaken as a specified frame. Also, the contents of a predetermined rulemay be, for example, that a frame of a time period in which a monitoringobject performing abnormal movement appears is taken as a specifiedframe. Whether or not movement of a monitoring object is abnormal can bedetermined, for example, by analyzing the behavior of a monitoringobject photographed in moving image data.

Here, monitoring object specification section 405 generates a monitoringobject specification screen based on monitoring object positionmanagement information stored in monitoring object position informationstorage section 404, and displays this screen on a liquid crystaldisplay. A monitoring object specification screen here is a screen forselection of a specified monitoring object and specified frame by auser.

FIG. 8 is a drawing showing a first example of a monitoring objectspecification screen.

As shown in FIG. 8, monitoring object specification section 405 displaysmonitoring object ID 643 of each monitoring object 642 in monitoringobject specification screen 640, superimposed on representativeomnidirectional image 641 stored in moving image storage section 402.Monitoring object ID 643 of each monitoring object 642 is displayed inmonitoring object specification screen 640 associated with the in-screenposition of that monitoring object 642. Monitoring object specificationsection 405 also displays message 644 prompting the user to select amonitoring object 642 in monitoring object specification screen 640.

Monitoring object specification section 405 may also display otheridentification information such as the name of a monitoring object(worker name) associated with a monitoring object ID in monitoringobject specification screen 640 together with, or instead of, monitoringobject ID 643.

When a monitoring object 642 the user wishes to be displayed inmonitoring object specification screen 640 is selected, monitoringobject specification section 405 acquires monitoring object ID 643 ofthat monitoring object 642 as a monitoring object ID of a specifiedmonitoring object. Then monitoring object specification section 405identifies a frame of an interval in which the specified monitoringobject has been photographed based on monitoring object positionmanagement information, and acquires the identified frame as a specifiedframe.

By means of monitoring object specification screen 640 of this kind, auser confirms an image, monitoring object ID 643, and so forth, of eachmonitoring object 642 in omnidirectional image 641, and selects aspecified monitoring object arbitrarily. That is to say, a user canselect each frame of an interval in which a selected specifiedmonitoring object is photographed as a specified frame.

FIG. 9 is a drawing showing a second example of a monitoring objectspecification screen.

As shown in FIG. 9, monitoring object specification section 405 displaysinformation of each monitoring object in monitoring object specificationscreen 650, superimposed on omnidirectional image 651 of monitored area200. Monitoring object information includes flow line 652 of eachmonitoring object created based on monitoring object position managementinformation, and monitoring object ID 653 associated with each flow line652. Flow line 652, for example, indicates a position at which amonitoring object enters monitored area 200 by means of a round end, andindicates a position at which a monitoring object leaves monitored area200 by means of an arrow-shaped end. Monitoring object specificationsection 405 also displays message 654 prompting the user to select aflow line 652 in monitoring object specification screen 650.

When a flow line 652 the user wishes to be displayed in monitoringobject specification screen 650 is selected, monitoring objectspecification section 405 acquires monitoring object ID 643 of that flowline 652 as a monitoring object ID of a specified monitoring object.Then monitoring object specification section 405 identifies a frame ofan interval in which the specified monitoring object has beenphotographed based on monitoring object position management information,and acquires the identified frame as a specified frame. An interval inwhich a specified monitoring object has been photographed is an intervalfrom entering to leaving of monitored area 200 by the specifiedmonitoring object.

By means of monitoring object specification screen 650 of this kind, auser can confirm places passed through by respective monitoring objects,and select a specified monitoring object arbitrarily. Then the user canselect each frame of an interval in which a selected specifiedmonitoring object is photographed as a specified frame. By this means,for example, in the event of an accident within monitored area 200 it iseasy to confirm a worker who passed through the site of the accident,and the movements of that worker, after the accident has occurred.

FIG. 10 is a drawing showing a third example of a monitoring objectspecification screen.

As shown in FIG. 10, monitoring object specification section 405displays paired frame ID 661 and monitoring object ID 662, for example,in monitoring object specification screen 660. Monitoring objectspecification section 405 also displays message 663 prompting the userto select one of the pairs in monitoring object specification screen660.

When a pair the user wishes to be displayed in monitoring objectspecification screen 660 is selected, monitoring object specificationsection 405 acquires that frame ID 661/monitoring object ID 662 pair asa monitoring object ID of a specified monitoring object and a frame IDof a specified frame.

By means of monitoring object specification screen 660 of this kind, auser can set a specified monitoring object and specified frame directlyby means of a monitoring object ID and frame ID. Then monitoring objectspecification section 405 identifies a frame of an interval in which thespecified monitoring object has been photographed subsequent to thespecified frame based on monitoring object position managementinformation, and acquires an identified frame as a specified frame. Oncompletion of step S1000, one specified monitoring object and aspecified frame (one or more consecutive frames) in which the specifiedmonitoring object has been photographed have been selected.

A user may wish to continue monitoring the same monitoring object evenwhen new moving image data is input. Considering such a case, provisionmay be made to allow monitoring object specification section 405 toreceive only a frame selection in step S1000, and to take the previouslyselected monitoring object as a specified monitoring object when only aframe selection is received.

Also, conversely, a user may wish to switch the monitoring object forthe same specified frame. Considering such a case, provision may be madeto allow monitoring object specification section 405 to receive only amonitoring object selection in step S1000, and to take a previouslyselected frame as a specified frame when only a monitoring objectselection is received.

In step S2000, monitoring object specification section 405 selects aframe with the smallest frame ID number among specified frames on whichprocessing has not been performed. Then monitoring object specificationsection 405 reports the frame ID of a currently selected specified frame(hereinafter referred to as “currently selected frame ID”) to movingimage storage section 402, and outputs an omnidirectional image of onespecified frame to image rotating section 406.

Also, monitoring object specification section 405 acquires the in-screenposition of a specified monitoring object (specified monitoring objectposition) in an omnidirectional image indicated by the currentlyselected frame ID from monitoring object position management informationstored in monitoring object position information storage section 404(see FIG. 6). Then monitoring object specification section 405 outputsthe acquired specified monitoring object position to image rotatingsection 406.

In the following description, it is assumed that a monitoring objecthaving monitoring object ID 643 “11” has been selected as a specifiedmonitoring object in monitoring object specification screen 640 shown inFIG. 8.

In step S3000, image rotating section 406 performs image rotationprocessing on an omnidirectional image input from moving image storagesection 402, based on a specified monitoring object position input frommonitoring object specification section 405. Here, image rotatingsection 406 generates an image in which monitoring object havingmonitoring object ID “11” imaged aslant at the upper-right in monitoringobject specification screen 640 shown in FIG. 8 is positioned in thecenter of the upper part of the image.

FIG. 11 is a flowchart showing an example of image rotation processing.

First, in step S3100, image rotating section 406 extracts an image areaoccupied by a specified monitoring object within an omnidirectionalimage (here, a monitoring object having monitoring object ID “11”). Forexample, image rotating section 406 extracts a monitoring object area bymeans of a background differencing technique in the vicinity of thespecified monitoring object position of a specified monitoring object,and extracts a monitoring object image area by means of templatematching using the shape of a monitoring object (for example, a person)as a template.

When monitoring object position information input section 403 acquiresmonitoring object position coordinates by means of image recognition,image rotating section 406 may define a specified monitoring object areausing a monitoring object image area extracted at that time.

FIG. 12 is a drawing showing an example of a specified frameomnidirectional image in initial-state.

As shown in FIG. 12, specified frame omnidirectional image ininitial-state (pre-rotation image) 670 is a circular downward-view imagein the same way as omnidirectional image 641 of monitoring objectspecification screen 640 shown in FIG. 8, and is an image whose centeris directly below an omnidirectional camera in real space. Therefore,direction 673 corresponding to the vertical direction in real space ofimage 672 of a specified monitoring object located at the upper-right isinclined at an angle.

In FIG. 12, square moving image data 674 defines the size ofomnidirectional image 670 and monitoring object positions and imageareas in the omnidirectional image, with the bottom-left corner as thexy coordinate system origin (0, 0), the rightward direction as thex-axis direction, and the upward direction as the y-axis direction.

If the radius of omnidirectional image 670 is designated r, upperendpoint, lower endpoint, right endpoint, and left endpoint coordinatesof the omnidirectional image are (r, 2r), (r, 0), (2r, r), and (0, r),respectively. The coordinate values shown in FIG. 12 are examples for acase in which r=500.

Then, in step S3200, image rotating section 406 defines a specifiedmonitoring object area based on an image area extracted in step S3100,and generates monitoring object position information that includes aspecified monitoring object position and specified monitoring objectarea.

FIG. 13 is a drawing showing an example of definition of a specifiedmonitoring object area, and FIG. 14 is a drawing showing an example ofmonitoring object area information.

As shown in FIG. 13, specified monitoring object area 680 is, forexample, an area that circumscribes specified monitoring object imagearea 681, and is enclosed by an isosceles trapezoid whose non-parallelopposite sides 682 and 683 each coincide with a radial direction ofomnidirectional image 684.

As shown in FIG. 14, image rotating section 406 can represent aspecified monitoring object area by means of the xy coordinates of threeendpoints of an isosceles trapezoid. Monitoring object area coordinates(upper-left) 701 are the coordinates of left endpoint 687 of the longerof parallel opposite sides 685 and 686 of the isosceles trapezoid (theside nearer the circumference). Monitoring object area coordinates(upper-right) 702 are the coordinates of right endpoint 688 of thelonger of parallel opposite sides 685 and 686 of the isoscelestrapezoid. Monitoring object area coordinates (lower-left) 703 are thecoordinates of left endpoint 689 of the shorter of parallel oppositesides 685 and 686 of the isosceles trapezoid (the side nearer theorigin).

Then image rotating section 406 calculates a transformation matrix forgenerating a post-rotation image from a pre-rotation image, based onacquired specified monitoring object position 690.

Specifically, this is done as follows, for example. First, imagerotating section 406 acquires specified monitoring object position 690for a pre-rotation image, and takes angle 693 formed by reference line691 and monitoring object line 692 as rotation amount θ ofomnidirectional image 684. Here, reference line 691 is a line extendingin an upward direction (y-axis positive direction) from center 694 ofomnidirectional image 684 when viewing omnidirectional image 684, andmonitoring object line 692 is a line linking center 694 ofomnidirectional image 684 and specified monitoring object position 690.Monitoring object line 692 coincides, for example, with a line linkingcenter 694 of omnidirectional image 684 and the midpoint betweenmonitoring object area coordinates (upper-left) 687 and monitoringobject area coordinates (upper-right) 688.

FIG. 15 is a drawing for explaining an example of a rotation amountcalculation method, and corresponds to FIG. 14. Parts in FIG. 15identical to those in FIG. 14 are assigned the same reference codes asin FIG. 14, and descriptions thereof are omitted here. In the followingdescription, a counterclockwise rotation direction about center 694 isconsidered to be positive.

In FIG. 15, when the coordinates of midpoint 695 between monitoringobject area coordinates (upper-left) 687 and monitoring object areacoordinates (upper-right) 688 are designated (xm, ym), monitoring objectline 692 passing through midpoint 695 and center 694 is represented byequation 1 below.

$\begin{matrix}{\left( {{Equation}\mspace{14mu} 1} \right)\mspace{616mu}} & \; \\{y = {{\left( \frac{y_{m} - r}{x_{m} - r} \right)x} + \left( {r - \frac{y_{m} - r}{x_{m} - r}} \right)}} & \lbrack 1\rbrack\end{matrix}$

Here, the point of intersection of perpendicular 698 dropped from pointof intersection 697 of reference line 691 and the circumference ofomnidirectional image 684 to monitoring object line 692 and monitoringobject line 692 is designated point of intersection 699. At this time,if the distance between point of intersection 697 and point ofintersection 699 is designated d1 and the distance between center 694and point of intersection 699 is designated d2, rotation amount θ can becalculated from equations 2 and 3 below.

$\begin{matrix}{\left( {{Equation}\mspace{14mu} 2} \right)\mspace{619mu}} & \; \\{{\sin \; \theta} = \frac{d_{1}}{r}} & \lbrack 2\rbrack \\{\left( {{Equation}{\mspace{11mu} \;}3} \right)\mspace{616mu}} & \; \\{{\cos \; \theta} = \frac{d_{2}}{r}} & \lbrack 3\rbrack\end{matrix}$

When coordinates in a pre-rotation image are designated (x, y) andcoordinates in a post-rotation image are designated (x′, y′), acoefficient part corresponding to coordinates (x, y) in a pre-rotationimage shown in the right-hand term of equation 4 below is calculated asa transformation matrix. Using this transformation matrix, imagerotating section 406 can generate a post-rotation image in which aspecified monitoring object is positioned in the center of the upperpart of the image.

$\begin{matrix}{\left( {{Equation}\mspace{14mu} 4} \right)\mspace{616mu}} & \; \\\begin{matrix}{\begin{pmatrix}x^{\prime} \\y^{\prime} \\1\end{pmatrix} = {\begin{pmatrix}1 & 0 & r \\0 & 1 & r \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}{\cos \; \theta} & {{- \sin}\; \theta} & 0 \\{\sin \; \theta} & {\cos \; \theta} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & {- r} \\0 & 1 & {- r} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}}} \\{= {\begin{pmatrix}{\cos \; \theta} & {{- \sin}\; \theta} & {r - {r\; \cos \; \theta} + {r\; \sin \; \theta}} \\{\sin \; \theta} & {\cos \; \theta} & {r - {r\; \sin \; \theta} - {r\; \cos \; \theta}} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}}}\end{matrix} & \lbrack 4\rbrack\end{matrix}$

Then, in step S3400, image rotating section 406 uses the transformationmatrix calculated in step S3300 to convert pre-rotation imagecoordinates and generate a post-rotation image. Image rotating section406 then outputs the generated post-rotation image and monitoring objectposition information indicating a specified monitoring object positionand specified monitoring object area in the post-rotation image tocenter position moving section 407.

FIG. 16 is a drawing showing an example of a post-rotation image afterimage rotation processing has been performed on pre-rotation image 670shown in FIG. 11.

As shown in FIG. 16, in post-rotation image 670 a, specified monitoringobject image 672 is positioned in the upper center, and direction 673corresponding to the vertical direction in that real space has become adownward direction.

Next, in step S4000 in FIG. 7, center position moving section 407performs center position movement processing on a post-rotation image(pre-movement image) input from image rotating section 406.

When performing warping processing that moves the center position of apost-rotation image downward, center position moving section 407performs warping processing such that a specified monitoring object areais particularly enlarged.

FIG. 17 is a flowchart showing an example of center position movementprocessing.

First, in step S4100, center position moving section 407 divides apre-movement image area.

FIG. 18 is a drawing showing an example of pre-movement image areadivision.

Center position moving section 407, for example, divides pre-movementimage area 711 by means of a plurality of lines 714 extending radiallyfrom center 712 of pre-movement image area 711 toward circumference 713,and a plurality of circles 715 concentric with circumference 713 of thepre-movement image. When the omnidirectional image shown in FIG. 16 isinput, the coordinates of center 712 are (500, 500). Plurality of lines714 include at least line 714 a extending in an upward direction (y-axispositive direction) from 712, and comprise, for example, a predeterminednumber of equally spaced lines. Plurality of circles 715 are positioned,for example, so that for a predetermined number of circles the radialdifference from another circle 715 is fixed.

Then, in step S4200, center position moving section 407 decides aprojection destination coordinates of the center of the pre-movementimage. Center position moving section 407 decides center projectiondestination coordinates as downward coordinates (having a smallery-coordinate). A projection destination y-coordinate may be apredetermined value, or may be a value obtained by multiplying anoriginal y-coordinate by a predetermined ratio. For the omnidirectionalimage shown in FIG. 16, coordinates (500, 270) are decided, for example.

Center position moving section 407 also decides projection destinationcoordinates of points of intersection 720 through 724 of line 714 aextending upward from center 712 to circumference 713 and circles 715.Center position moving section 407 may make point of intersection 720 ofline 714 a and circumference 713 a fixed value, for example. Here, inFIG. 18, the distance between point of intersection 720 and point ofintersection 721 is radial direction interval 725, the distance betweenpoint of intersection 721 and point of intersection 722 is radialdirection interval 726, and the distance between point of intersection722 and point of intersection 723 is radial direction interval 727.Also, in FIG. 18, the distance between point of intersection 723 andpoint of intersection 724 is radial direction interval 728, and thedistance between point of intersection 724 and point of intersection 728is radial direction interval 729. At this time, center position movingsection 407 decides the mutual ratios of post-projection radialdirection intervals 725 through 729 (725: 726: 727: 728: 729). Forexample, center position moving section 407 decides the mutual ratios ofpost-projection radial direction intervals 725 through 729 (725: 726:727: 728: 729) as (1.0: 1.5: 0.8: 0.5: 0.2). The mutual ratios ofpost-projection radial direction intervals 725 through 729 are referredto below as “post-projection interval ratios.” Here, center positionmoving section 407 decides post-projection interval ratios so that aradial direction interval corresponding to a specified monitoring objectarea (here, radial direction interval 726) becomes wider so thatenlarged display of the specified monitoring object is performed. Centerposition moving section 407 may also decide a radial direction intervalcorresponding to a specified monitoring object area (here, radialdirection interval 726) using a ratio for the distance between center712 and point of intersection 721 (for example, 0.3).

A center position movement distance due to projection and apost-projection interval ratio should preferably be values such that thedetailed situation of a specified monitoring object and the ambientsituation are sufficiently visible, and are decided based onexperimentation or experience, for example.

A monitoring object positioned farther from omnidirectional camera 300(that is, nearer omnidirectional image circumference 713) is displayedas smaller in a pre-movement image. Therefore, center position movingsection 407 may decide post-projection interval ratios so that otherradial direction intervals are progressively larger toward thecircumference while enlarging the radial direction interval of a partcorresponding to a specified monitoring object area.

Also, center position moving section 407 may increase the ratio of apart corresponding to a specified monitoring object area the longer thedistance between omnidirectional camera 300 and a specified monitoringobject. In this case, for example, center position moving section 407stores beforehand a correspondence relationship between each position inan omnidirectional image and the distance to a monitoring object imagedat that position (the distance to the floor on which that monitoringobject is standing). Then center position moving section 407 acquires adistance corresponding to a specified monitoring object position as adistance between omnidirectional camera 300 and the specified monitoringobject. Center position moving section 407 then decides the ratio of apart corresponding to a specified monitoring object area based on theacquired distance.

Then, in step S4300, center position moving section 407 divides thepost-movement image area. Specifically, center position moving section407 decides each straight line and curved line dividing thepost-movement image area as linking projection destination coordinatepoints at post-movement coordinates for which coordinates were decidedin step S4200. By this means, the way in which straight lines and curvedlines dividing the post-movement image area are projected onto thepost-movement image area is decided.

FIG. 19 is a drawing showing an example of post-movement image areadivision.

As shown in FIG. 19, center position moving section 407 positionsdecided post-projection center 733, for example. Then center positionmoving section 407 divides line 734 extending upward from center 733based on the decided post-projection interval ratios, and positionsdivision points 735 through 738. Center position moving section 407 alsodivides line 734′ extending downward from center 733 based on thedecided post-projection interval ratios, and positions division points735′ through 738′.

Following this, as shown in FIG. 19, center position moving section 407divides post-movement image area 731 with ellipses 739 passing throughpositioned division points 735′ through 738′, corresponding respectivelyto pre-movement image circles (circles 715 in FIG. 18). Furthermore,center position moving section 407 divides post-movement image area 731with lines 740 such as to divide the circumferences of ellipses 739 atequal intervals, corresponding to lines in the pre-movement image (lines714 in FIG. 18).

Then, in step S4400, center position moving section 407 performs aprojective transformation from the division areas of the pre-movementimage to corresponding division areas of the post-movement image. Forexample, center position moving section 407 performs a projectivetransformation of an image of division area 741 in the pre-movementimage to corresponding division area 742 in the post-movement image.

Each division area is a quadrilateral or triangle having curved lines assides, but in order to simplify the processing, center position movingsection 407 may perform projective transformation regarding all sides asstraight lines. Also, the smaller the area of a pre-movement imagedivision area and post-movement image division area, the less noticeableare the seams between division areas in the post-movement image, and thecleaner is the image.

Center position moving section 407 outputs a generated post-movementimage, and monitoring object position information indicating a specifiedmonitoring object position and specified monitoring object area in thepost-movement image, to ellipsification section 408.

Next, in step S5000 in FIG. 7, ellipsification section 408 performsellipsification processing on a post-movement image (pre-ellipsificationimage) input from center position moving section 407. Specifically,ellipsification section 408 executes the two-dimensional matrixtransformation shown in equation 5 below, for example, on apre-ellipsification image. Here, (x′, y′) indicates coordinates in apre-ellipsification image, and (x″, y″) indicates coordinates in apost-ellipsification image. Sx indicates an x-axis direction enlargementfactor when performing ellipsification.

$\begin{matrix}{\left( {{Equation}\mspace{14mu} 5} \right)\mspace{619mu}} & \; \\{\left( \frac{x^{''}}{y^{''}} \right) = {\begin{pmatrix}{Sx} & 0 \\0 & 0\end{pmatrix}\left( \frac{x^{\prime}}{y^{\prime}} \right)}} & \lbrack 5\rbrack\end{matrix}$

Enlargement factor Sx may be a fixed value, or may be decided based onthe size or aspect ratio of a specified object area. For example,ellipsification section 408 may calculate the aspect ratio of aspecified monitoring object area, and employ an enlargement factor Sxsuch that that aspect ratio approaches a general aspect ratio when aperson is viewed from the side. Here, the enlargement factor is assumedto be 1.5.

FIG. 20 is a drawing showing an example of the state in apost-ellipsification image of divided areas in a post-movement image.

As shown in FIG. 20, for pre-ellipsification image center 733 and line734 extended upward from the center, the x-coordinate value is enlargedby the decided enlargement factor and is moved in the x-axis positivedirection. Also, pre-ellipsification image circumference 732 andellipses 739 are extended in the x-axis direction by the decidedenlargement factor. Together with this, ellipsification section 408 alsoextends lines 740 dividing the circumferences of ellipses 739 at equalintervals, and division area 742, in the x-axis direction.

Ellipsification section 408 outputs a generated post-ellipsificationimage, and monitoring object position information indicating a specifiedmonitoring object position and specified monitoring object area in thepost-ellipsification image, to distortion correction section 409.

Next, in step S6000 in FIG. 7, distortion correction section 409performs distortion correction processing on a post-ellipsificationimage (pre-distortion-correction image) input from ellipsificationsection 408. Specifically, distortion correction section 409 performscorrection of distortion such as rounding of straight linescharacteristic of an omnidirectional camera 300 photographed image,which is difficult to resolve with the above-described contentmanipulation processing and ellipsification processing.

FIG. 21 is a flowchart showing an example of distortion correctionprocessing.

First, in step S6100, distortion correction section 409 acquires aspecified monitoring object area in a post-ellipsification image.Specifically, distortion correction section 409 acquires a specifiedmonitoring object area in a pre-distortion-correction image(post-ellipsification image) input from ellipsification section 408.

Then, in step S6200, distortion correction section 409 executesdistortion correction on a calculated specified monitoring object area.The contents of distortion correction are in accordance with the opticalcharacteristics and so forth of omnidirectional camera 300, and aredetermined based on experimentation or experience, for example.Distortion correction section 409 may use a distortion correctionfunction in general image processing software. Depending on the contentsof distortion correction, the size and shape of a specified monitoringobject area may not be changed.

Then, in step S6300, distortion correction section 409 determineswhether or not there is a change in at least either the size or shape ofa specified monitoring object area. If there is a change in at leasteither the size or shape of the specified monitoring object area (S6300:YES), distortion correction section 409 proceeds to step S6400. If thereis no change in either the size or shape of the specified monitoringobject area (S6300: NO), distortion correction section 409 returnsdirectly to the processing in FIG. 7.

In step S6400, distortion correction section 409 performs projectivetransformation of an area other than a specified monitoring object areaof an omnidirectional image in a form that matches a change in the sizeand shape of the specified monitoring object area to the surroundingarea.

FIG. 22 is a drawing showing an example of pre-distortion-correctionimage area division, and FIG. 23 is a drawing showing an example ofpost-distortion-correction image area division.

Here, it is assumed that specified monitoring object area 762 inpre-distortion-correction image 761 corresponds to four division areas(the hatched parts in the drawing), as shown in FIG. 22. Division areadelimiting is not necessarily the same as for a specified managementarea, but in order to simplify the description, a specified monitoringobject area is here assumed to be composed of a plurality of divisionareas. It is also assumed that the optical characteristics ofomnidirectional camera 300 are such that, in pre-distortion-correctionimage 76, distortion makes the horizontal direction progressivelynarrower toward center 763 of pre-distortion-correction image 761.

In this case, distortion correction section 409 performs correction onpre-distortion-correction image 761 so that specified monitoring objectarea 762 widens more toward center 763 of pre-distortion-correctionimage 761. Furthermore, distortion correction section 409 performscorrection on pre-distortion-correction image 761 so that a specifiedmonitoring object area for which distortion has been resolved isdisplayed as larger.

As a result, specified monitoring object area 772 ofpost-distortion-correction image 771 shown in FIG. 23 has distortionresolved to a greater extent than in the case of specified monitoringobject area 762 prior to distortion correction shown in FIG. 22 througha change in the ratio of the upper side to the lower side of anisosceles trapezoid. Moreover, specified monitoring object area 772 ofpost-distortion-correction image 771 is larger in area than specifiedmonitoring object area 762 prior to distortion correction.

Also, distortion correction section 409 corrects the size and shape ofdivision areas adjacent to specified monitoring object area 772 so thatthe adjacency relationship between division areas is maintained. At thistime, distortion correction section 409 corrects the size and shape ofeach division area so that a change in specified monitoring object area772 is absorbed gradually from division area 774 adjacent to specifiedmonitoring object area 772 to division area 775 farther away.

Then distortion correction section 409 performs projectivetransformation of each part other than a specified monitoring objectarea of the omnidirectional image to a corresponding division area aftercorrection, and generates a post-distortion-correction image composed ofa specified monitoring object area and other areas after distortioncorrection. Distortion correction section 409 then outputs the generatedpost-distortion-correction image to output image storage section 410.

By this means, a clean omnidirectional image can be obtained in whichseams between a specified monitoring object area and its surroundingsare not noticeable, even when distortion correction changes the size andshape of a specified monitoring object area.

Output image storage section 410 stores a post-distortion-correctionimage input from distortion correction section 409, and outputs thepost-distortion-correction image to image output section 411.

Next, in step S7000 in FIG. 7, image output section 411 displays apost-distortion-correction image input from output image storage section410 by means of a display such as a liquid crystal display.

Then, in step S8000, monitoring object specification section 405determines whether or not there is a specified frame that has not beenselected in step S2000.

If the frame is not the last of the specified frames—that is, if thereis an unselected specified frame (S8000: NO), monitoring objectspecification section 405 returns to step S2000. That is to say,monitoring object specification section 405 increments the currentlyselected frame ID by 1 and selects the next specified frame, and repeatssteps S2000 through S8000.

If the frame is the last of the specified frames—that is, if there is nounselected specified frame (S8000: YES), monitoring object specificationsection 405 returns to step S2000. That is to say, monitoring objectspecification section 405 terminates the series of processing steps.

By means of this kind of operation, omnidirectional image processingapparatus 400 can receive a specification of a monitoring object that auser wishes to focus on in omnidirectional image video, and can performimage processing on video so that that monitoring object becomes easy tosee. Also, since only warping processing and linear projectivetransformation are performed after image rotation, the continuity of anomnidirectional image can be maintained.

Also, when there are a plurality of specified frames, omnidirectionalimage processing apparatus 400 can track the movement of auser-specified monitoring object within monitored area 200, andcontinuously display a relevant monitoring object in the upper centerpart of an omnidirectional image at all times.

As described above, an omnidirectional image processing apparatusaccording to this embodiment performs image processing on anomnidirectional image so that directions corresponding to a verticaldirection and horizontal direction in real space of a specifiedmonitoring object approach a downward direction and horizontal directionwhen viewing the omnidirectional image. By this means, a post-processingomnidirectional image can be displayed in a state closer to a state whena specified monitoring object is actually viewed. That is to say, apost-processing omnidirectional image can be displayed in a state inwhich the head of a specified monitoring object is above and the feetare below, and distortion of horizontal components is further reduced.Also, since the surroundings of a specified monitoring object areextended vertically, a specified monitoring object and ambient imagescan be displayed as larger, and an omnidirectional image can bepresented that enables the situation of a specified monitoring objectand the surroundings thereof to be more easily grasped.

Since an omnidirectional image processing apparatus according to thisembodiment performs the above-described image processing by means ofomnidirectional image rotation and warping processing that moves thecenter position of an omnidirectional image, the continuity of anomnidirectional image can be maintained.

An omnidirectional image processing apparatus according to thisembodiment performs warping processing such that a specified monitoringobject area is further enlarged when the center position of anomnidirectional image is moved. Furthermore, an omnidirectional imageprocessing apparatus according to this embodiment performsellipsification processing that extends an omnidirectional imagelaterally by means of linear projective transformation. By means ofthese processing operations, a post-processing omnidirectional imagedisplays a specified monitoring object and a surrounding image as stilllarger, enabling the situation of a specified monitoring object and thesurroundings thereof to be more easily grasped.

An omnidirectional image processing apparatus according to thisembodiment performs warping processing such that distortion of aspecified monitoring object area is corrected, and a surrounding imagearea is corrected in line with that correction. By this means, apost-processing omnidirectional image enables the situation of aspecified monitoring object to be more easily grasped.

In this embodiment, a case has been described in which an informationstorage operation is completed for a collection of moving images (aplurality of specified frames) prior to an image processing operation,but application of the present invention is not limited to this.Provision may also be made, for example, for an omnidirectional imageprocessing apparatus to perform an information storage operation andimage processing operation on a frame-by-frame basis, and perform imageprocessing and display of an omnidirectional image in real time. In thiscase, it is necessary for a specified frame decision to be made based ona currently displayed video specification, or based on whether or notthere is a specified monitoring object, instead of being based on aframe ID specification.

An omnidirectional image processing apparatus need not necessarilyexecute all of the above-described image rotation processing, specifiedobject area enlargement in center position movement processing,ellipsification processing, and distortion correction processing. Forexample, a specified monitoring object may be approaching theomnidirectional camera, and image quality may be good, with littledistortion. In such a case, the omnidirectional image processingapparatus can improve the visibility of a monitoring object specified inthat omnidirectional image while maintaining omnidirectional imagecontinuity simply by executing only image rotation processing and centerposition movement processing.

In this embodiment, a specified monitoring object area has beendescribed as an area enclosed by a frame circumscribing the outline of aperson, but application of the present invention is not limited to this.A specified monitoring object area may be an area that includes not onlya specified monitoring object area but also another object, such as anobject located near that specified monitoring object and related to thebehavior of that specified object (for example, a workbench—hereinafterreferred to as “ancillary object”).

FIG. 24 is a drawing showing another example of definition of aspecified monitoring object area.

As shown in FIG. 24, it is assumed that specified monitoring object 781works while viewing display 782. In this case, an omnidirectional imageprocessing apparatus, for example, uses a line of sight detectionsection (not shown) to detect the line of sight of the worker who isspecified monitoring object 781 by means of image recognition such astracking pupil movement. The omnidirectional image processing apparatusthen determines display 782 to be an ancillary monitoring object basedon the detected line of sight. The line of sight detection section canbe implemented by means of a camera or the like installed on a desk atwhich specified monitoring object 781 works, or at a low position on awall. Then, for example, image rotating section 406 decides upon an areaenclosed by the shape of an isosceles trapezoid that circumscribes anoutline including both the outline of specified monitoring object 781and display 782 that is an ancillary object. By using an area thatincludes a specified monitoring object and an ancillary monitoringobject as a specified monitoring object area in this way, the state ofan ancillary object can also be displayed in an easy-to-see fashion, andthe behavior of a specified monitoring object can be grasped moreeasily.

A specified monitoring object area need not be an above-describedisosceles trapezoid, but may also be of another shape, such asrectangular or elliptical, as long as its position in an omnidirectionalimage can be appropriately defined. Also, in this embodiment, provisionmay also be made for a monitoring object image area to be displayed inthe monitoring object specification screen in FIG. 8 after undergoingdistortion correction.

In this embodiment, a case has been described in which an initial-stateomnidirectional image is a downward-view image, but application of thepresent invention is not limited to this. For example, application ofthe present invention is also possible when an initial-stateomnidirectional image is an upward-view image. An upward-view image is,for example, an image photographed by installing omnidirectional camera300 on the floor of monitored area 200, facing the ceiling.

As described above, in a downward-view image, in an image of a workerstanding on the floor of monitored area 200, the head is oriented towardthe circumference of an omnidirectional image, and the lower body isoriented toward the center of the omnidirectional image. In contrast, inan upward-view image, in an image of a worker standing on the floor ofmonitored area 200, the head is oriented toward the center of anomnidirectional image, and the lower body is oriented toward thecircumference of the omnidirectional image.

Therefore, when an initial-state omnidirectional image is an upward-viewimage, an omnidirectional image processing apparatus should rotate aspecified monitoring object to the lower center in image rotationprocessing, and move the center of the omnidirectional image upward incenter position movement processing.

In this embodiment, a case has been described in which anomnidirectional camera is used that has a small reflecting mirrorinstalled on the incident light axis, but it is also possible to applythe present invention to a case in which an omnidirectional camera thatemploys a fisheye lens is used.

FIG. 25 is a drawing showing an example of a specified frameomnidirectional image in initial-state when a fisheye lens is employed,and corresponds to FIG. 12.

As shown in FIG. 25, when an omnidirectional camera that employs afisheye lens is used, a blind area 504 does not occur in the center ofpre-rotation image 670, which is an initial-state omnidirectional image.Therefore, the presence of monitoring object 784 positioned at thecenter point of pre-rotation image 670 is possible.

When monitoring object 784 of this kind is present, an omnidirectionalimage processing apparatus performs image rotation processing, forexample, so that the vertical direction in real space of monitoringobject 784 coincides with the downward direction when an omnidirectionalimage is viewed.

FIG. 26 is a drawing showing an example of a post-rotation image afterimage rotation processing has been performed on pre-rotation image 670shown in FIG. 25 relative to monitoring object 784, and corresponds toFIG. 16. By means of this processing, post-rotation image 670 a can beobtained in which a monitoring object nearest an omnidirectional camera(directly below an omnidirectional camera) is clearly visible.

It is possible for an omnidirectional image processing apparatus toimprove the visibility of a monitoring object by performing thefollowing kinds of processing, for example, after image rotationprocessing. The first is processing that enlarges a monitoring objectimage area while keeping a monitoring object at the center point of anomnidirectional image. The second is processing that moves the center ofan omnidirectional image upward, and moves the center of anomnidirectional image obtained as a result downward. According to thesecond processing, a monitoring object is displayed in the upper part ofan omnidirectional image, and it is therefore desirable to furtherperform distortion correction processing as necessary.

In this embodiment, a monitoring system for monitoring workers in afactory has been described, but application of the present invention isnot limited to this. The present invention can be applied to variouskinds of apparatuses and systems that handle omnidirectional images,such as a system that photographs the interior of a store in order tomonitor shoplifting activity, for example.

The disclosure of Japanese Patent Application No. 2010-62560, filed onMar. 18, 2010, including the specification, drawings and abstract, isincorporated herein by reference in its entirety.

INDUSTRIAL APPLICABILITY

An omnidirectional image processing apparatus and omnidirectional imageprocessing method according to the present invention enable thevisibility of a specified monitoring object in an omnidirectional imageto be improved while maintaining the continuity of that omnidirectionalimage, and are therefore suitable for use as a monitoring system thatmonitors a wide area such as a store or factory.

REFERENCE SIGNS LIST

-   100 Monitoring system-   200 Monitored area-   300 Omnidirectional camera-   400 Omnidirectional image processing apparatus-   401 Moving image input section-   402 Moving image storage section-   403 Monitoring object position information input section-   404 Monitoring object position information storage section-   405 Monitoring object specification section-   406 Image rotating section-   407 Center position moving section-   408 Ellipsification section-   409 Distortion correction section-   410 Output image storage section-   411 Image output section

1. An omnidirectional image processing apparatus that performs imageprocessing on an omnidirectional image, the omnidirectional imageprocessing apparatus comprising: a monitoring object specificationsection that specifies a monitoring object photographed in theomnidirectional image; an image rotating section that rotates theomnidirectional image so that a position of the specified monitoringobject becomes upper-central; and a center position moving section thatmoves a center position of the omnidirectional image rotated by theimage rotating section downward by means of warping processing.
 2. Theomnidirectional image processing apparatus according to claim 1, furthercomprising a monitoring object position information input section thatinputs a position of the specified monitoring object in theomnidirectional image in initial-state, wherein the image rotatingsection rotates the omnidirectional image based on the position of thespecified monitoring object.
 3. The omnidirectional image processingapparatus according to claim 1, wherein the center position movingsection enlarges an image area of the specified monitoring object. 4.The omnidirectional image processing apparatus according to claim 1,further comprising an image ellipsification section that changes a shapeof the omnidirectional image to an elliptical shape by extending theomnidirectional image horizontally.
 5. The omnidirectional imageprocessing apparatus according to claim 1, further comprising adistortion correction section that locally changes a shape of an imagearea of the specified monitoring object among monitoring objectsphotographed in the omnidirectional image, and corrects distortionthereof.
 6. The omnidirectional image processing apparatus according toclaim 2, further comprising: a moving image storage section that storesa moving image that is time series data of the omnidirectional image; amonitoring object position information storage section that stores aposition of the specified monitoring object at each time input by themonitoring object position information input section; and an imageoutput section that outputs the moving image on which the imageprocessing has been performed.
 7. An omnidirectional image processingmethod that performs image processing on an omnidirectional image, theomnidirectional image processing method comprising: a step of specifyinga monitoring object photographed in the omnidirectional image; a step ofrotating the omnidirectional image so that a position of the specifiedmonitoring object becomes upper-central; and a step of moving a centerposition of rotated the omnidirectional image downward by means ofwarping processing.